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Abstract 

By reviewing the literature and interviewing 83 individuals knowledgeable about massive open online 
courses (MOOCs), we investigate the goals of institutions of higher education that are currently 
developing and delivering such courses. We identify six major goals for MOOC initiatives: extending 
reach and access, building and maintaining brand, improving economics by reducing costs or increasing 
revenues, improving educational outcomes, innovation in teaching and learning, and conducting research 
on teaching and learning. Comparing these goals with the data being collected about MOOCs, their 
participants, and educational outcomes, as well as the resource requirements and cost drivers of the 
development and delivery process, we assess whether these goals are being met, or are likely to be in the 
future. While quantification of success in achieving these goals is for the most part lacking, we conclude 
that institutions are experiencing at least partial success in achieving each of these goals except for 
improving economics. We identify obstacles to fuller achievement of the goals and some potential 
solutions. 


Introduction 

Over the past few years, observers of higher education have speculated about dramatic changes 
that must occur to accommodate more learners at lower costs and to facilitate a shift away from 
accumulation of knowledge to acquisition of a variety of cognitive and noncognitive skills (e.g., 
“Connecting the Dots,” 2013). All scenarios feature a major role for technology and online learning. 
Massive open online courses (MOOCs) are the most recent candidates being pushed forward to fulfill 
these ambitious goals. To date, there has been little evidence collected that would allow an assessment of 
whether MOOCs do indeed provide a cost-effective mechanism for producing desirable educational 
outcomes at scale. It is not even clear that these are the goals of the institutions offering MOOCs. This 
paper is a report on the actual goals of institutions creating MOOCs or integrating them into their 
programs and reviews the current evidence regarding whether and how these goals are being achieved. 

Online education at the college level has been expanding rapidly over the last decade with 
students participating in single courses or even earning entire degrees without setting foot in a brick-and- 
mortar institution. According to Allen and Seaman (2013, 2014), the online enrollment growth rate in the 
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United States has ranged between 6.1% and 36.5% per year since 2002. Means, Bakia, and Murphy 
(2014) outline and document four major trends in how universities are using online learning: “self-paced, 
adaptive instruction and competency-based learning; blended learning; learning analytics; and MOOCs” 
(p. 46). The arrival of MOOCs, which allow hundreds of thousands of students to participate 
simultaneously in a course and are free and open to any interested participant, constitutes a phenomenon 
that extends preexisting initiatives to provide free, educational resources online, such as MIT 
OpenCourseWare, Stanford Engineering Everywhere, and Khan Academy. 

The goals, structure, and pedagogical philosophy of connectivist MOOCs (cMOOCs), first 
offered by Siemens, Downes, and other Canadian instructors (see Downes, 2008; Cormier & Siemens, 
2010) and “xMOOCs”, first offered by Ng, Thrun, Norvig, and Widom at Stanford University (see 
Markoff, 2011; Waldrop, 2013) are radically different, with the only commonality being that they are 
scalable and technology based. Whereas in cMOOCs learning occurs through participant interactions with 
a network of individuals and participants who are expected to create, share, and build upon each other’s 
artifacts (e.g., videos, blog posts), xMOOCs are primarily designed to deliver education at scale and 
involve more structured and sequenced direct transmission of knowledge. There has been little agreement 
as to what actually constitutes a MOOC and what educational or other objectives they can and should 
address. However, as observed by Lewin (2013), one universal impact of MOOCs is clear: “The intense 
publicity about MOOCs has nudged almost every university toward developing an Internet strategy” 
(para. 11). Allen and Seaman (2014) report that, in 2013, 5% of 2,831 U.S. institutions responding to an 
annual survey about online learning were offering a MOOC, 9% were planning to do so, and 53% were 
undecided as to whether to engage in this innovation. Larger institutions were more likely to offer a 
MOOC than smaller ones, as were doctoral/research institutions compared with institutions offering less 
advanced degrees. The value and purposes of engaging with MOOCs have been less clear, and many 
undecided institutions are struggling with whether and how to join the race to provide education using 
this relatively untested method of pedagogical delivery. At the same time, development and delivery of 
MOOCs requires a significant commitment of personnel time and institutional resources. Hollands and 
Tirthali (2014) document that MOOC production and delivery almost invariably involve multiple actors 
contributing several hundred hours of their time, costing their institutions between $39,000 and $325,000 
per MOOC. Clearly, this demand on effort and resources needs to be justified in terms of expected 
benefits. 

It is curious that MOOCs have taken hold without much evidence as to whether they are effective 
in improving participants’ skills and knowledge, or in addressing other objectives, and without evidence 
of their economic value. As Means et al. (2014) observe, “Both irrational exuberance and deep-seated fear 
concerning online learning are running high” (p. 42). This study seeks to address the lack of clarity 
surrounding institutional motivations for developing and delivering MOOCs and to investigate whether 
MOOCs are worth the considerable resources and effort being expended on their creation. We first review 
what is known about institutional goals for pursuing MOOCs. Subsequently, we describe our methods for 
interviewing a variety of stakeholders in the MOOCscape. We summarize our findings in terms of six 
goal areas, in each case assessing whether the goals are being met, what the obstacles to their realization 
are, and how they could be better fulfilled in the future. 

Literature Review 

Peer-reviewed publications on MOOCs first began appearing in 2008, (see review by 
Liyanagunawardena, Adams, & Williams, 2013), and some early indicators of student performance in 
MOOCs have emerged over the past two years (e.g., Breslow et al., 2013; Firmin, Schiorring, Whitmer, 
Willett, & Sujitparapitaya, 2013; Champaign et al., 2014), but there is a dearth of rigorous studies 
investigating the effectiveness of MOOCs in addressing educational objectives. Researchers are still 
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formulating ideas about how to assess the effectiveness of different types of MOOCs. Some researchers 
(e.g., Grover, Franz, Schneider, & Pea, 2013; DeBoer, Ho, Stump, & Breslow, 2014) point out that 
current discussions about the effectiveness of MOOCs are based on assumptions and outcome variables 
relevant to previous learning environments, and that there is a need to reffame the discussions and 
reconceptualize the variables. In addition, MOOCs are being pursued at many institutions for reasons 
other than the improvement of teaching and learning, creating the need for alternative metrics to assess 
the impact of MOOCs on these other objectives. 

Hollands and Tirthali (2014) report that colleges and universities have adopted several different 
stances toward engaging with MOOCs. Some are actively developing MOOCs and may be termed 
“producers,” some are using MOOCs developed by other institutions in their programs and could be 
termed “consumers,” and a few are doing both. Others are adopting a wait-and-see approach, and some 
have considered MOOCs and have either decided against any form of official engagement or have not 
met with interest from faculty members to pursue them. While many people assume that MOOC 
production is limited to “elite” universities, other institutions, including a few community colleges, have 
concluded that they know how to serve their particular students best and have created MOOCs tailored to 
the needs of their own student populations. Consumers of MOOCs are integrating MOOCs created by 
other institutions into their course offerings in flipped classrooms (e.g., San Jose State University, see 
Cheal, 2012) or simply as supplemental resources for their students. Instructors at the University System 
of Maryland are experimenting with a variety of approaches to embedding MOOCs created by others into 
their classes (Griffiths, 2013; Ithaka S+R, 2013; Ithaka S+R, 2014). A small number of institutions have 
declared willingness to consider MOOCs for credit (e.g., Georgia State University, University of 
Maryland University College). A few universities, such as Vanderbilt University and the University of 
California, Berkeley, could be categorized as both producers and consumers, with some instructors 
creating MOOCs and others using MOOCs created by their own or other institutions in their classes. 

Allen and Seaman (2014) report that, in 2013, 5% of institutions responding to their survey of 
online education had already offered a MOOC. Among the 140 or so MOOC-offering institutions, 27% 
indicated that their primary objective was to increase the institution’s visibility, 20% primarily sought to 
increase student recruitment, 18% wanted to innovate pedagogy, and 17% wished to provide flexible 
learning opportunities. Other primary objectives, mentioned by fewer than 10 institutions each, included 
reaching new students, supplementing on-campus experiences, exploring cost reductions, learning about 
scaling, and creating revenues. Surprisingly, increasing the quality of teaching and learning did not appear 
to be a priority. As the survey asked only for the primary objective, it is possible that increasing the 
quality of teaching and learning may have been a secondary objective for some. Two thirds of the 
institutions felt it was too early to judge whether their objectives were being met, while one third felt 
some or all objectives were being met. It is not, however, clear whether these perceptions were based on 
substantive evidence. 

The Alliance for Higher Education and Democracy (AHEAD) at the University of Pennsylvania 
recently conducted a poll of administrators, faculty members, and other personnel at institutions of higher 
education (IHEs) in the United States and found that, among the approximately 44 respondents (reported 
as “around 29%” [p.4] of 153 respondents) at institutions offering a MOOC, 57% strongly agreed that 
MOOCs may be a potentially effective mechanism for “raising institutional profile” (AHEAD, 2014). 
Fifty percent of the respondents strongly agreed that MOOCs could help improve access around the 
globe, 40% that MOOCs can help improve access in the United States, 34% that MOOCs can improve 
pedagogy, and only 19% that they can reduce costs. Confidence in the potential for MOOCs to reduce 
costs was higher among respondents from institutions that had not yet offered MOOCs. For all other goal 
options offered by AHEAD, MOOC offerers appeared more confident than MOOC abstainers about the 
potential for MOOCs to achieve the goals. 
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This limited information from the Allen and Seaman and AHEAD surveys on the goals of 
institutions offering MOOCs suggests a variety of motivations that extend well beyond the dissemination 
of knowledge at scale. Our study was designed to investigate institutional goals and objectives for 
engaging with MOOCs through in-depth interviews, and to collect evidence of whether and how these 
goals are being achieved. 


Method 

Using a methodology similar to that employed by Bacow, Bowen, Guthrie, Lack, and Long (2012) to 
investigate barriers to online learning in higher education, we conducted a qualitative study (see Merriam, 
2009) comprising interviews of 83 individuals across a range of institutions. With the aim of eliciting 
perspectives on MOOCs from several angles, we sought representation from both public and private 
IHEs, both two-year and four-year institutions, researchers, online learning platform providers, other for- 
profit education companies, and several additional stakeholders in the online learning space. Table 1 
shows the distribution of interviewees across institutional types. 

Table 1 Affiliations of Interviewees 


Type of institution 

Number of institutions 
represented* 

Number of 
interviewees 

Public university 

16 

20 

Private university 

14 

26 

Community college 

9 

10 

Research organization 

7 

8 

Platform provider 

5 

6 

Other for-profit education company 

4 

5 

Other institution** 

7 

8 

Total 

62 

83 


*At the majority of institutions one person was interviewed, but at a few institutions several individuals were 
interviewed to include interviewees from multiple areas (for example, administrators, faculty members, and 
researchers). 

**Other institutions consisted of the following: one museum (two interviewees); one K-12 school district; one 
educational technology advocacy group; one higher education association; one venture capital firm; one private 
foundation; and one independent consultant. 

We contacted by e-mail individuals who appeared to be knowledgeable about MOOCs or online 
learning based on their position in deciding whether and how to participate in the MOOCspace, their 
experience teaching or planning MOOCs, or their writing and research in this area. Interviewees were 
identified from the existing academic and journalistic literature on MOOCs, by reviewing the names of 
conference presenters and panelists for sessions on MOOCs or online learning in higher education, by 
researching the MOOC activities of institutions on the Internet, and by consulting with known experts in 
the field of educational technology. Additionally, many of our interviewees suggested other people for us 
to interview either at their own institutions or elsewhere. Given the targeted nature of our recruitment 
process, our sample is not representative of U.S. educational institutions as a whole. However, in addition 
to stratifying our sample across public and private four-year institutions and community colleges, we 
aimed to include institutions and interviewees that were supportive of MOOCs and those that were not. 
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We contacted a total of 100 individuals on a rolling basis at 66 different institutions, 39 of which 
were colleges or universities. Of the 100 contacted, 83 people at 62 institutions were successfully 
interviewed within the time frame of our study. At least one person from every college or university we 
contacted agreed to participate. At most institutions we interviewed one person, but at a few institutions 
we interviewed two or more. Most interviewees were based in the United States, two were in China, two 
in the United Kingdom, and several were in Canada. Thirty of our interviewees were administrators at 
IHEs; 22 were faculty members, of which 7 also had administrative duties; 16 were executives at other 
institutions; 13 were researchers; one was an educational technologist; and one was a program officer at a 
foundation. 

Interviews were conducted between June 2013 and February 2014, and follow up by e-mail with 
interviewees to obtain updates and to verify information continued until May 2014. Slightly under half of 
the interviews were conducted face-to-face, and the others were conducted by telephone or Skype. 
Interviews lasted an average of 75 minutes and followed a semistructured interview protocol (Merriam, 
2009). Interview questions addressed what the interviewee’s institution was currently doing with respect 
to MOOCs; institutional goals for pursuing MOOCs; the interviewee’s role in the process; how the 
interviewee and his or her institution define a MOOC; the characteristics of MOOCs, including structure, 
purpose, enrollment level, fees, and credentials offered; educational objectives of specific MOOCs being 
offered; educational outcomes being measured; data being collected before, after, and during MOOC 
delivery; how MOOC-based research is being used to improve pedagogy on-campus; personnel and other 
resource requirements; cost drivers; how costs of MOOC production compare with face-to-face courses; 
how the interviewee sees the MOOC phenomenon developing over the next five years, and how this 
development might help the institution meet its ongoing goals. 

Most interviewees granted permission for the interview to be recorded, and the resulting digital 
audio files were transcribed. All interview notes and transcriptions were coded in NVivo software using 
deductive codes initially derived from the interview protocol. These initial codes were supplemented by 
inductive codes that were iteratively refined as more granular themes were identified (LeCompte & 
Schensul, 1999). Interviewee perspectives were supplemented with a review of the MOOC-related 
literature and perspectives offered by presenters at various academically oriented MOOC-related events. 
A comprehensive report of interview findings was circulated to all interviewees in April 2014 to allow for 
corrections and clarifications. This paper focuses on the data relevant to the goals of institutions engaging 
with MOOCs. 


Results 

Of the 62 institutions that participated in this study, 29 had already offered a MOOC or had used 
one or more MOOCs in their programs, or were in the midst of doing so. Our interviewees identified a 
variety of institutional goals, which fell into one of six categories: 

• extending the reach of the institution and access to education 

• building and maintaining brand 

• improving economics by lowering costs or increasing revenues 

• improving educational outcomes for MOOC participants and on-campus students 

• innovation in teaching and learning 

• conducting research on teaching and learning 

Table 2 indicates the number of each type of educational institution in our sample at which one or 
more interviewees mentioned each goal, as well as the percentage of all 83 interviewees who mentioned 
each goal. 
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Table 2 Institutional Goals for Developing and Delivering or Using MOOCs 


Institutional goal 

Number of institutions offering/using MOOCs stating 
this as a goal 

% of all 

. interviewees who 


% of total 

0 n = 29)* 

Private 
universities 
(n = 10) 

Public 
universities 
(n = 15) 

Community 

colleges 

0 = 3) 

raised this as a 
goal 
(i n = 83) 

Extending reach and access 

65 

7 

8 

3 

42 

Building and maintaining 
brand 

41 

3 

8 

0 

25 

Improving economics 

38 

2 

8 

1 

29 

Improving educational 
outcomes 

38 

4 

5 

2 

20 

Innovation in teaching and 
learning 

38 

3 

7 

1 

19 

Research on teaching and 
learning 

28 

4 

3 

1 

18 


* Includes one museum in addition to the universities and colleges 


Goal 1: Extending Reach and Access to Education Extending the reach of the institution to a wider 
audience and improving access to education was the most commonly identified goal for offering a 
MOOC, mentioned by 65% of the institutions in our study that were offering or using MOOCs and 42% 
of our interviewees overall. These interviewees included administrators and faculty members from seven 
private universities, eight public universities, three community colleges, and a museum. Some institutions 
claimed to be pursuing an altruistic goal of reaching the masses globally with high-quality educational 
experiences, while others presented more defined goals of reaching a specific population or solving a 
particular challenge related to access. Several instructors described being motivated to offer MOOCs 
because of their passion for their subject and desire to make it accessible to the general public. Ways in 
which MOOCs are expected to increase access to education include the following: 

• “broadcasting” to global audiences 

• alleviating infrastructure constraints domestically and in rapidly developing 
countries where the existing physical campus infrastructure and level of faculty 
expertise cannot accommodate the growing demand for postsecondary education 

• easing the pressure on oversubscribed programs or “bottleneck” courses 

• providing flexibility in time and place of study for nontraditional students 

• providing a no-risk, low-cost option for at-risk students in developmental 
education, setting them on an accelerated path to credit-bearing courses and more 
timely completion of a degree 

• increasing access to instructors skilled in specialized domains and niche subjects; 

• flexibility for students to create their own programs using courses from various 
institutions 

• continuing education or professional development for alumni and other working 
adults 

Evidence of MOOCs Extending Reach and Access Institutions offering MOOCs via the major MOOC 
platforms have attracted a substantial number of participants: edX reported around two million unique 
users as of March 2014, and Coursera reported over seven million. In January 2014, Coursera reported 
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over 22 million enrollments, with many participants enrolling in multiple courses. These enrollees were 
spread across 571 courses and were based in 190 different countries (see 
https://www.coursera.org/about/conmiunity) . While these numbers are impressive, only a small fraction 
of those who registered actually participated in the courses, and far fewer completed them. Ho et al. 
(2014), using data from 17 HarvardX and MITx MOOCs offered between fall 2012 and summer 2013, 
found that 841,687 people registered for the MOOCs, but 35% of them never engaged with the content, 
and 56% engaged with less than half of it. Around 43,000, or 5% of participants, completed their courses. 
A typical registrant was “male with a bachelor’s degree who is 26 or older” (p. 2). This finding is in line 
with Christensen et al.’s (2013) claim that 79% of MOOC participants have a bachelor's degree, based on 
a survey of participants in 24 MOOCs offered on the Coursera platform by the University of 
Pennsylvania. With respect to global reach, 72% of the registrants in the HarvardX and MITx MOOCs 
were from outside the United States, and 2.7% were from countries on the United Nations’ list of Least 
Developed Countries (Ho et al., 2014). Among the 34,779 participants in the University of Pennsylvania 
sample, 31% were from non-U.S., OECD countries; 15% were from BR1CS (Brazil, Russia, India, China, 
and South Africa); and 20% were from other developing countries. 

These data suggest that MOOCs are providing educational opportunities to millions of individuals 
across the world. However, most are already well educated, and only a small fraction of these participants 
fully engages with the courses. Among our interviewees, we found mixed views as to whether MOOCs 
have succeeded in improving access to their institutions’ offerings and reaching wider audiences. Some 
had no doubt that MOOCs had already expanded access to their courses while others acknowledged that 
the initial expectations that MOOCs would “democratize” education had not been realized and that 
MOOCs may even be doing more to increase gaps in access to education than to diminish them. 
Quantifying the impact of MOOCs offered by a particular institution on improving access to its regular 
courses is difficult unless institutions accurately document the level and diversity of participation in their 
programs both before and after offering MOOCs. For the most part, institutions were not yet making a 
concerted effort to document these potential changes. 

Barriers to Using MOOCs to Extend Reach and Access, and Possible Solutions Several interviewees 
pointed out that simply providing free resources does not necessarily make them accessible to everyone. 
The end user may still experience infrastructure constraints and cultural barriers. In some locations 
Internet bandwidth is inadequate for MOOC participation, which involves downloading materials and 
watching videos. Buying additional bandwidth is very expensive in some countries, such as China, and 
may be prohibitive for many would-be MOOC participants. Adaptations have been made in certain 
countries to overcome bandwidth limitations. For example, according to one interviewee, in Pakistan, 
where the infrastructure outside of cities cannot support MOOCs, the government is partnering with 
FutureLearn and Coursera to transfer MOOC content to CDs. ZayaLabs, a technology startup, provides 
tablets and a router loaded with MOOC content in a backpack to schools in India, Indonesia, and 
Mongolia. Taking the content offline allows participants to engage with it in the absence of any wireless 
infrastructure. As platform providers such as NextThought and Coursera move toward mobile access to 
their content, some of these infrastructure constraints may be alleviated. 

Less easily addressed were some pessimistic views about the capacity of online courses, massive 
ones in particular, to reach any but the most dedicated students. One interviewee observed that access to 
educational resources is only one part of the equation to guarantee success for the majority of college 
students, with the need to motivate students, track their progress, and guide them along a trajectory 
toward completion being equally important. Another observed that education is more complex than 
simply providing access to content. In addition to logistical and pedagogical barriers to improving access 
to education through MOOCs, a number of regulatory obstacles were mentioned, especially for 
community colleges and K-12 schools. Such obstacles will only be resolved if accreditation agencies and 
state education departments review and revise regulations to accommodate online offerings that reach far 
beyond local confines. 
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Goal 2: Building and Maintaining Brand For IFIEs, building and maintaining brand serves to attract 
and retain students, faculty members, and partnership opportunities with other institutions, funders, and 
alumni networks. Interviewees from 41% of the institutions that were offering or using MOOCs stated 
that branding, positioning, or attracting students was a strategic goal for the initiative. These interviewees 
represented seven public universities, three private universities, and a museum. None of the community 
colleges mentioned branding as a goal for engaging in MOOCs. Branding involves different strategies for 
different types of institutions, depending on their current positioning. For flagship state universities in the 
United States that have already established local appeal, MOOCs allow for more national and 
international recognition. For elite institutions, being at the forefront of a highly publicized innovation 
serves as a signaling mechanism to protect their global ranking among the top universities. Community 
colleges were less concerned about branding than improving access to education for an underprepared 
student population. 

Institutions that were employing MOOCs as a vehicle to expand their brand made strategic 
decisions to showcase programs, specialties, and research capabilities in which they believed they were 
leaders or wanted to be among the first to stake a claim to world-class expertise. Some institutions 
perceived building brand as a direct pathway to higher recruitment and enrollment in tuition-earning 
courses and programs. MOOCs offer participants a “frictionless”—that is, a low-risk, no-cost— 
introduction to a program area and the offering institution can simultaneously identify candidates who 
might perform well as degree-earning students. In addition to recruiting students, some institutions 
emphasized the need to showcase themselves as the leader in academic content areas and in pedagogical 
innovation so as to attract and retain the best faculty members. MOOCs also offer an opportunity to 
maintain connections to alumni, who are an important source of funding. Finally, MOOCs provide 
opportunities to forge mutually beneficial partnerships with other educational institutions, and with 
corporate partners and funding agencies, potentially leading to increased grant revenues. 

While many institutions have received significant media attention as a result of their MOOC 
activities, isolating and measuring the impact of any new initiative on brand is a difficult exercise. Most 
institutions are only beginning to think about how to capture and quantify branding-related benefits—for 
example, by comparing historical data on applications and admissions with post-MOOC statistics. One 
interviewee from an elite institution observed that selective institutions must balance the apparently 
contradictory goals of increasing access to their offerings with building and maintaining brand. Another 
complication raised was the question of where the brand actually lies. Do participants opt for a course 
because it is on a particular platform or because it is offered by a particular university? It was noted that 
as MOOC platform providers such as edX and Coursera open their platforms to a wider range of 
institutions, some of the initial cachet of belonging to these consortia is being lost. 

Goal 3: Improving Economics: Reducing Costs or Increasing Revenues Thirty-eight percent of the 
institutions in our sample that were offering or using MOOCs, and 29% of our interviewees overall, 
claimed that a goal for their MOOC initiatives was to lower costs or increase revenues, or both. Among 
the 29 institutions currently offering or using MOOCs, this goal was mentioned by representatives of 
eight of the 15 public, four-year universities, two of the ten private, four-year universities, and one of the 
three community colleges, suggesting that it was more important to public universities than private ones. 
Many interviewees expressed concern at the lack of sustainability for MOOC initiatives given the heavy 
burden on faculty time and other institutional resources. There was widespread acknowledgement that the 
current expenditures on MOOC development could not continue indefinitely without financial 
justification. 
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Potential Cost Savings from MOOCs To date, MOOC production and delivery have resulted in a 
significant drain on time and money for institutions, but interviewees offered several possibilities for 
eventual cost savings: 

• reusing MOOC materials multiple times 

• sharing MOOC materials across instructors and campuses 

• developing common courses to offer across institutions 

• replacing on-campus courses with MOOCs 

• faculty time savings 

• reducing the need for facilities 

• recruitment efficiencies 

• less costly student support services provided by nonfaculty members 

• increasing student throughput 

Not surprisingly given the infancy of MOOC initiatives, we were able to identify very few 
examples of actual costs savings realized. To the contrary, we were able to gather significant evidence of 
the high costs of development and delivery of MOOCs. Realizing cost and time savings may be more 
difficult than anticipated. Some interviewees, based on their experiences with regular online courses, 
questioned the likelihood of being able to reuse MOOC materials without significant adjustments each 
time. Others observed that sharing MOOCs across campuses or developing common courses would 
require the resolution of logistical issues such as how tuition and credits are transferred, and a shift in 
culture away from creation of content at each site. Few interviewees believed that MOOCs could fully 
substitute for on-campus instruction, expecting that they would need to be accompanied by face-to-face 
interactions between a live instructor and the students. 

One example we encountered where less costly student support services are being provided is the 
MOOC-based Online Master of Science in Computer Science (OMS CS) program that the Georgia 
Institute of Technology (Georgia Tech) is developing with Udacity. Each course is staffed by a Georgia 
Tech instructor and a Head Teaching Assistant (TA) serving a projected 127 students in Year 1, rising to 
456 students in Year 3 (although the actual average course size for the five courses offered in spring 2014 
was 115 students). The Head TA, while selected by Georgia Tech, is employed by Udacity. In addition, 
Udacity-based Course Managers approved by Georgia Tech each provide technical support to around 390 
students. Georgia Tech graduate students, serving 40 students each, serve as TAs to help with grading and 
responding to academic questions. Georgia Tech and Udacity are working on ways to manage the 
workload as enrollments increase, including more use of automated grading. Courses are initially taught 
by a faculty member, but reruns may be taught by a nontenured instructor or the Head TA. While this 
model may rely less than regular programs on expensive faculty time, it is not clear yet whether it will be 
sufficient to adequately support students’ needs. 

MOOCs as a Source of Revenue A small number of interviewees speculated that some IHEs are 
pursuing MOOCs as a potential source of revenue, but only two representatives from state universities 
explicitly stated revenue generation as a goal. Potential sources of revenue included the following: 

• offering credit for MOOC completion and charging tuition 

• creating new for-fee courses and programs 

• drawing MOOC participants into existing, full-tuition degree programs 

• increasing class sizes 

• licensing fees for use of MOOC materials or data by other institutions 

• fees for additional services offered to MOOC participants—for example, online tutoring 
or face-to-face instruction with a local instructor 


9 





Why do Institutions Offer MOOCs? 


• increased grant revenues 

• matchmaking or training for employers 

There are a few instances in which MOOCs or MOOC-like courses have been offered for credit 
with tuition being charged. San Jose State University piloted online courses that were offered free and 
without credit to the public, but a smaller number of formally enrolled students paid $150 each to earn 
three to five credits per course. Georgia Tech’s OMS CS program charges $134 per credit. In each 
instance where MOOC-like courses are offered for credit, the offering institution has provided some 
degree of supplementary face-to-face instruction or online support for course participants. However, 
based on our review of the financial and enrollment projections for Georgia Tech’s OMS CS program, we 
do not expect that the revenues will exceed the high fixed costs of MOOC development over the initial 
three-year period. We did not have access to financial projections for San Jose State University’s MOOC 
initiative that would allow a similar review. At the University of Oklahoma, 20 credit-bearing, online 
courses have been developed with on-campus students as the primary audience in a flipped classroom 
model. These courses carry the same credits as the university’s traditional courses and charge the same 
tuition. The courses are also open as online only, tuition- and credit-free MOOCs to any member of the 
public. 


Evidence to support the likelihood of revenues materializing from other sources was more 
limited. We heard several anecdotal reports of increased interest in an institution’s regular courses 
subsequent to a MOOC offering, but we found only one documented instance in which MOOC 
participants were being drawn into an existing, full-tuition program. University of Texas at Arlington’s 
School of Nursing offered a MOOC through MOOC2Degree, and an undisclosed number of participants 
were in the process of applying to the university’s program. Several interviewees claimed that using 
MOOC materials to flip the classroom and automating certain aspects of on-campus courses had allowed 
more students to enroll in a class, but it was unclear whether there was a net increase in number of 
students at the university or whether these students were simply being drawn away from other existing 
courses and programs. Some fees have been earned by offering MOOC participants verified certificates of 
completion, but the percentage uptake for this service has, to date, only been on the order of 1 % of initial 
enrollees. While we heard several reports of licensing arrangements being signed to allow one institution 
to use another’s MOOC content, we encountered no instances where fees had actually changed hands. 
Revenue increases from grants or from matchmaking or training services for employers currently 
appeared to be purely aspirational. 

Goal 4: Improving Educational Outcomes Thirty-eight percent of the institutions participating in our 
study that were offering or using MOOCs and 20% of our interviewees overall expected MOOCs to lead 
to an improvement in educational outcomes, some believing improvement would occur directly within the 
MOOC format and some others believing improvement would happen indirectly through the transfer of 
new strategies and techniques to on-campus teaching. This goal was mentioned by representatives of five 
public universities, four private universities, and two community colleges. Ways in which MOOCs were 
expected to lead to improvements in educational outcomes included the following: 

• providing instant feedback to course participants 

• gamification and badging to increase motivation 

• outreach to MOOC participants to encourage persistence 

• adaptive learning, personalization, or mastery-based learning 

• flipping the classroom using MOOCs to provide the online content 

• motivating instructors to rethink pedagogy 

• redesigning regular courses to incorporate MOOC strategies, such as “chunking” lectures 
and interspersing questions, and increasing opportunities for peer-to-peer learning 
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• using MOOCs in K-12 to prepare students for college 

• fine-tuning instructional materials 

According to a number of interviewees, the most significant impact of MOOCs has been on the 
motivation they have created for instructors to rethink how they teach. Several interviewees reported that 
MOOCs have prompted even typically intransigent faculty members to reconsider their teaching styles 
and have facilitated faculty professional development that might otherwise be resisted. However, several 
interviewees remarked on the lack of evidence that better learning experiences are actually being created 
for students, observing that change in practice is not inherently positive. Indeed, while interviewees 
provided many examples of how MOOCs have been used to change instruction, for the most part, actual 
impact on educational outcomes has not been documented in any rigorous fashion. Consequently, in most 
cases, it is unclear whether the goal of improving educational outcomes has been achieved. Cima (2014) 
and Ghadiri, Qayoumi, Junn, Hsu, and Sujitparapitaya (2013) provide two exceptions in which 
instructional changes and their effects on student performance have been carefully recorded. While not 
gold-standard, random-assignment experiments, they provide promising evidence of improvement in 
student performance as a result of integrating MOOCs into flipped on-campus courses, or adopting some 
of the MOOC strategies, such as frequent assessment and automatic feedback. On the other hand, Ithaka 
S+R (2014) conducted several side-by-side comparison tests of courses at the University System of 
Maryland and found no significant differences in posttest gains between students participating in hybrid 
courses that included MOOC materials and those participating in traditional versions of the courses. 

While many interviewees pointed to the potential for MOOCs to contribute significantly to the 
development of personalized and adaptive learning, it is clear that current turnaround time for analyzing 
MOOC platform data is too long to allow for even midcourse corrections, let alone to provide just-in-time 
customization for individual participants. Without exception, interviewees acknowledged that this 
potential is far from being realized. A great deal of coordination and collaboration among content experts, 
instructors, researchers, instructional designers, and programmers will be necessary to result in 
substantive progress towards personalized and adaptive learning. 

Goal 5: Innovation in Teaching and Learning Thirty-eight percent of the institutions in our study that 
were engaging with MOOCs and 19% of our interviewees overall presented MOOCs as vehicles for 
experimenting with and innovating pedagogy and models of higher education. Innovation as an end goal 
was mentioned by representatives of seven public universities, three private universities, and one 
community college. We note, however, that the other five goals we report involve innovating as a means 
to a particular end. 

A number of interviewees portrayed experimentation with MOOCs and online learning as 
preparation for an uncertain future. Institutions do not want to be left behind in what they perceive could 
be a game-changing phenomenon for higher education. MOOCs were also presented as a disruptive 
innovation within the higher education business model that could help universities become more 
competitive simply by forcing them to reconsider the status quo and to ask fundamental questions about 
commonly accepted practices. One interviewee asserted that MOOCs have pushed college personnel to 
open up to the research about how people learn. A senior administrator observed that, especially for large 
state universities, change cannot be implemented swiftly. If these institutions do not at least experiment 
with innovative ideas as they arise, they will always lag behind smaller, more nimble competitors. A 
researcher at one of the pioneering MOOC producers described a trajectory of innovation in MOOCs 
starting with the initial infatuation with scale, proceeding to the search for a peer-mediated solution to the 
challenge of grading thousands of assignments, to the current recognition that participants may learn best 
when connected with other participants and finding ways to facilitate this through online chat rooms and 
small group discussions. 
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Not all of our interviewees were as convinced that MOOCs would introduce more productive 
innovation than existing experiments with online and blended learning. Interviewees with experience in 
online education and open learning resources over the last two decades pointed out that many MOOC 
developers are reinventing and relearning the missteps and successes of online learning and failing to take 
advantage of scale, the most characteristic aspect of MOOCs that differentiates them from other online 
education. 

The large number of courses that have been developed or substantively redesigned since the 
appearance of MOOCs can be considered as evidence of innovation—for example, MOOC.list currently 
catalogs around 1,700 courses. It is abundantly clear that MOOCs have prompted many institutions and 
faculty members to engage in new educational activities. The strategies employed online, such as frequent 
assessments and short lectures interspersed with questions, are subsequently being used on campus. It is 
less clear what has been gained by these new initiatives because the value of innovation is hard to 
measure unless it can be tied to a more tangible objective. While a few institutions are developing metrics 
to assess the impact of MOOC-related innovations on various objectives, most institutions are not yet 
making any rigorous attempt to assess whether MOOCs are more or less effective than other strategies to 
achieve their goals. 

Goal 6: Research on Teaching and Learning Research on teaching and learning was stated as a goal 
for MOOC initiatives by 28% of the 29 institutions offering or using MOOCs and by 18% of our 83 
interviewees. This was a stated goal for one community college, three public research universities, and 
four private research universities. There is considerable overlap between formal research and ad hoc 
efforts to improve teaching and learning. While there is often a fine line of distinction between Goal 4 and 
Goal 6, in this section we generally address work that is being conducted by individuals who consider 
themselves researchers first and foremost and aim to publish their work, while efforts described under 
Goal 4 were less about creating generalizable knowledge and more about improving educational 
outcomes in a specific situation. 

We heard from our interviewees about several types of research being conducted that use a 
MOOC as the vehicle for delivering an intervention or as a source of data on participant behavior and 
performance. The researchers themselves were in a range of disciplines, which influenced the kinds of 
questions being asked. Computer scientists and engineers have tended to focus either on descriptive 
studies, summarizing demographics or establishing how much each educational resource (video, e-text, 
discussion forum, etc.) was used, or on design research to create badging systems, discussion 
opportunities, peer grading, and auto-grading applications. Sociologists and education (or learning 
science) researchers have more often focused on experimental research—for example conducting A/B 
tests by e-mailing different messages or materials to different participants selected at random. Areas in 
which interviewees described research explorations and advances included the following: 

• the role of social media networks in teaching and learning 

• testing pedagogical strategies—for example, Firmin et al. (2013) and Champaign et al. 
(2014) 

• student engagement and motivation—for example, Coetzee, Fox, Hearst, and Hartmann 
(2014); Kizilcec, Piech, and Schneider (2013); and Kizilcec, Schneider, Cohen, and 
McFarland (2014) 

• machine learning and modeling research—for example, Piech et al. (2013); Halawa, 
Greene, and Mitchell (2014); and Nguyen, Piech, Huang, and Guibas (2014) 

• natural language processing 

• human-computer interaction—for example, Kulkami et al. (2013) and Cambre, Kulkami, 
Bernstein, and Klemmer (2014) 
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• personalized and adaptive learning—for example, Buffardi and Edwards (2014) 

• comparing hybrid with traditional courses—for example, Ithaka S+R (2013, 2014) 
Griffiths (2013); and Bruff, Fisher, McEwen, and Smith (2013) 

• developing data standards and a common platform for data mining—for example, 
Veeramachaneni, Demoncourt, Taylor, Pardos, and O'Reilly (2013) and Demoncourt et 
al. (2013) 

A great deal of effort has been expended on trying to improve participant engagement and 
completion of MOOCs and less effort on determining whether participants actually gain skills or 
knowledge from the courses. The majority of instructors and researchers appear content with equating the 
completion of a MOOC with having learned something. One exception is the work of RELATE at MIT 
(see Colvin et al., in press) which rigorously documents both absolute and relative learning in a physics 
MOOC using pre- and posttesting and Item Response Theory. A few interviewees noted that this question 
of efficacy is rarely asked with respect to regular college courses. Others questioned whether findings 
from MOOCs can be generalized to other populations and contexts. Many noted that progress in using 
MOOCs for educational research is being impeded by difficulty using the platform data and lack of clarity 
regarding regulations applicable to the participants and their data. Another obstacle is that in a MOOC 
researchers do not make the decisions about what gets taught and how, and faculty members may or may 
not be predisposed toward the researcher’s goals. 

Conclusions and Recommendations 

It appears that the goal of MOOC producers to extend reach to a wider audience and improve 
access to education has been met to some extent in terms of geographical spread but less so in terms of 
reaching individuals with fewer educational opportunities. Participants are mostly limited to highly 
motivated learners who have access to high-bandwidth Internet connections, and the majority have 
already earned at least a bachelor’s degree. These may be new audiences for many existing college and 
university courses and programs, but it appears that MOOCs are mostly educating the educated and are 
therefore increasing the divide between those who have access to education and those who do not. It may 
be more accurate to state that they are extending access to lifelong education rather than making 
education accessible to a broad group of people. On the other hand, this educated audience is also more 
likely to be able to pay for certificates and other services, allowing the production and delivery of 
MOOCs to be financially sustainable. While many institutions express the desire to improve access to 
education—and this is indubitably a laudable goal—there is little real incentive or financial justification 
for institutions to do so, except for those participants who can pay for it. 

Going forward, if institutions genuinely wish to broaden access to less educated audiences, they 
must identify multiple channels of communication to reach potential recruits. For example, they may need 
to use social media networks and advertise through high schools, employment agencies, or community 
organizations in the United States and abroad. Additionally, courses must be designed to serve learners 
who are less self-directed by incorporating motivational features and optional instructional scaffolding to 
address differences in participant preparation levels. However, as IHEs can ill afford to offer free and 
unsubsidized educational opportunities to the world, the costs of developing MOOCs will need to be 
covered by some fraction of participants who have adequate resources. 

If success in achieving the goal of building and maintaining brand was measured purely by media 
publicity surrounding MOOCs, many institutions have indeed become more visible as a result of their 
MOOC activities. This publicity has been mostly positive, but not always. Although several institutions 
suggested anecdotally that this goal has been partially accomplished, quantifying success could be 
achieved by comparing pre- and post-MOOC metrics on student recruitment and enrollment statistics, 
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success in faculty recruitment and retention, breadth and generosity of donors, and quantity and size of 
grants. In addition, the costs of MOOC production and delivery should be compared to other strategies 
known to be effective at building and maintaining brand in order to ascertain which alternatives are most 
cost-effective. 

Typical of any early-stage, technology-based innovation, achievement of the third goal, to 
improve the economics of higher education, is still elusive. We found scant evidence that MOOCs have 
increased revenues substantially and much evidence that they have increased costs significantly. 
Gradually materializing revenues could help shift the balance over the next few years, but unless MOOC 
producers and platform providers continue to expand ways to confer economic value on MOOC 
completion—for example, in the form of employer-recognized credentials—the market and associated 
revenue streams will remain limited. One way or another, MOOCs will need to prove they can solve a 
problem or provide a service more efficiently than existing alternatives. 

MOOCs could potentially reduce the costs of higher education if they were used to eliminate the 
reproduction of similar courses across many campuses. If the licensing issues surrounding such sharing of 
materials can be resolved, and if MOOCs can be rerun several times without significant adjustments at 
each offering, the high costs of initial MOOC development could be amortized over all the instances of 
use. However, as costs of delivering education are mostly personnel related, overall costs of higher 
education can only fall if personnel numbers or salaries are reduced. Reduction of costs could be realized 
if academic institutions were willing to shift more of the responsibility for instructional support away 
from tenured faculty members and onto nontenured instructors, teaching assistants, or outsourced 
personnel. Tenured faculty positions could not be easily eliminated in the short term, but they could be 
reduced over time, contributing to the current long-term trend towards “adjunctization” of higher 
education. This trend does not, however, bode well for the capacity of academic institutions to further 
various academic disciplines through research, nor is it likely to be welcomed by faculty members and 
their unions. For now, MOOCs are being promoted as a substitute for repetitive activities, such as 
delivering the same lecture every year or grading uniform assignments, while faculty members spend time 
in the classroom with students on problem solving, discussion, and debate. This approach may prove to be 
more educationally effective than current practice, but it will also be more expensive and therefore even 
more financially unsustainable. 

From the perspective of learners, MOOCs will only favorably impact the economics of obtaining 
an education if they remain free or charge minimal fees but can be used to substitute for costlier on- 
campus or online courses to obtain college credits or other credentials that are accepted by employers. 
While this scenario may be beneficial to the students and to the taxpayer, it may not be in the interests of 
the academic institutions themselves, as it will imply a loss of revenue per student, which may or may not 
be possible to make up by increasing enrollments. State and federal education policymakers could adjust 
regulations to create pathways for MOOCs to be accepted for credit in high schools or to satisfy 
government-mandated continuing education for professionals, but in higher education it is less clear how 
accreditation agencies could be persuaded to move in this direction. Establishment of an accrediting 
organization for individual MOOCs and other nontraditional educational experiences would allow 
learners to accumulate a portfolio of credentials that serve as a viable supplement or alternative to a 
college degree. Furthermore, such accreditation of individual courses or other nontraditional educational 
experiences should confer the ability to use public funds toward the costs of these credentials. Fain (2013) 
reports on two potential efforts in this direction and highlights some of the surrounding regulatory issues. 
Ironically, MOOCs have introduced possibilities that, should they materialize, could significantly 
compete with the very institutions that are racing to develop them. 
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To assess the sustainability of MOOCs, researchers and evaluators will need to document how 
much the costs of delivery fall for repeat offerings of MOOCs, how enrollment numbers change, and what 
the uptake rate is for any revenue-generating services. To determine whether MOOCs are a cost-effective 
means to deliver education or achieve other objectives, it will be necessary to compare the costs of 
MOOCs to the costs of alternative interventions, as well as the effectiveness of each alternative in 
addressing a common outcome of interest, such as increasing participants’ knowledge or skills. It would 
also be useful to gauge the reaction of employers to such nontraditional credentials and to inquire as to 
what modifications employers might require to allow these credentials to be considered in employment or 
promotion decisions. The more value employers are willing to place on these credentials, the more fees 
MOOC offerers can charge to participants in order to offset institutional costs of production and delivery. 

Many interviewees would readily assert that the fourth goal of improving educational outcomes 
has been achieved through MOOCs, at least for a limited group of learners. These positive reports arise in 
situations where MOOCs have been integrated with on-campus courses or where on-campus courses have 
been redesigned to incorporate MOOC-like components. Documentation of a small number of these 
experiments has shown early signs of improving student performance as measured by assessments and 
course completion rates. Typical strategies include flipping the classroom, implementing frequent 
assessments, and spending class time on problem-solving activities, often in small groups, rather than on 
lecture. 


Little effort has been made to ascertain whether participants in stand-alone MOOCs gain useful 
skills and knowledge that can be applied in productive, real-world contexts. To determine whether the 
goal of improving educational outcomes can indeed be achieved through MOOCs alone will require 
rigorous evaluation employing pre- and postassessments of knowledge and skills, and comparisons with 
the outcomes of face-to-face or other online courses. Educators must progress beyond the notion that 
course completion per se equates with learning and must consider how to allow learners to demonstrate 
valuable competencies, both cognitive and noncognitive, that may be acquired through MOOC 
participation. Longitudinal studies tracking post-MOOC outcomes—such as sequences of courses taken, 
professional certifications obtained, or job opportunities received—would help assess the longer term 
value of participating in MOOCs. More effort is also required to assess whether MOOCs as stand-alone 
experiences can be designed effectively to increase knowledge and cultivate skills for any but the most 
motivated learners. Future research could be designed to broaden the types of learners represented in 
studies of MOOC activity and impact in order to avoid the presentation of results that are not applicable 
to the majority of learners. 

The goal of innovation in teaching and learning has certainly been met at institutions where 
online learning did not previously exist, as evidenced by the sheer number of new courses produced and 
course redesign efforts undertaken. Metrics on the frequency of these efforts could be compared with pre- 
MOOC activity to provide an assessment of how well the goal of innovation has been achieved, but it is 
not clear that innovation per se is a sufficient aspiration. In many cases, institutions are innovating with a 
further goal in mind, such as improving educational outcomes or building brand. Assessing the impact of 
MOOC initiatives on these distal goals would be a more worthwhile endeavor in judging their 
contribution to higher education. Some institutions are unclear as to why they are embarking on MOOC 
initiatives and until they can agree internally on suitable and realistic goals, they will struggle to justify 
the expense and effort. For institutions that have been offering online and hybrid courses for many years, 
MOOCs represent more of an incremental step along a preexisting trajectory than a major innovation. 
Some even consider MOOCs to be regressive because they fail to incoiporate best practices in distance 
learning and repeat many of the mistakes of earlier attempts to educate at scale. Going forward, 
researchers and practitioners who are familiar with evidence-based best practices in online learning could 
usefully recommend pedagogical strategies that can be effectively migrated to MOOCs. Finally, the goal 
of conducting research on teaching and learning has encountered many obstacles, but a few well-funded 
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institutions with strong computer programming capacity have not been deterred from efforts to use the 
“big data” from MOOC delivery platforms to document online learner behavior and to experiment with 
variations in pedagogical strategies. Regrettably, MOOCs are more often serving as a playground 
populated by captive participants within which researchers tinker with the artifacts than as an object of 
research to determine whether MOOCs, compared with existing models of education, are an effective and 
cost-effective means of educating a broad range of learners. MOOC researchers could benefit from 
soliciting the involvement of experienced online learning practitioners or researchers as they plan studies 
to experiment with MOOCs. A potentially useful application of the big data from MOOCs is to provide 
learner analytics. These can be used to iteratively improve courses or to catalyze the advent of high- 
quality adaptive learning and personalized educational experiences that meet the needs of a variety of 
learners. It is apparent, however, that the useful application of data-mining techniques to data from 
MOOC platforms could be facilitated by standardization of data formats across the various platforms. It is 
also clear that the high demand on resources to develop sophisticated adaptive-learning mechanisms will 
require the establishment of working partnerships between educators, instructional designers, and 
programmers. Federal approval of cross-institutional Institutional Review Board agreements could 
facilitate projects in which expertise and resources can be pooled. Additionally, clarifications regarding 
the applicability of FERPA to MOOCs would alleviate current confusion over how to handle participant 
data. 


Given the considerable investment that MOOCs represent, we strongly recommend that prior to 
embarking on MOOC-related initiatives, institutions carefully consider their goals and whether MOOCs 
present a realistic and financially justifiable means to achieve them compared with existing alternatives. 
Administrators at academic institutions should work with representative faculty members and, where 
relevant, their unions to establish a strategy for MOOC engagement and a process for production or 
adoption of MOOCs. Data collection protocols should be devised up front to measure relevant indicators 
both prior to and following MOOC engagement in order to permit an objective assessment of whether 
stated goals are being met. Costs of such initiatives can be estimated by tracking personnel time 
commitments with respect to MOOC-related efforts and accounting for displacement of other productive 
activities, such as teaching or grant writing. These costs can be compared with the evidence of goal 
achievement to judge whether MOOCs are a worthwhile investment of limited educational resources. 
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